Subjectivity: Its Role in Exploratory Search Processes and Evaluation

Subjectivity is inherent in exploratory search. It is evident in the attributes of the searcher before beginning the search (e.g., the searcher’s domain knowledge). In the form of the searcher’s interests, knowledge, cognitive/learning style, and personality, it affects the search process – its overall trajectory. And it comes into play at the end of the search, when the searcher makes judgments about the success of the search. Researchers and designers interested in exploratory search will need to develop measures of searchers’ beliefs about a search’s success and attitudes toward the search outcomes. These measures can then be leveraged to more directly evaluate the quality of the search process and to design personalized systems that improve a searcher’s performance. INTRODUCTION As White et al. [32] note, “defining what constitutes an exploratory search is challenging” [p.38]. Nevertheless, they provide an initial definition, in concert with that provided by Marchionini [21], that, “in exploratory search, users generally combine querying and browsing strategies to foster learning and investigation” [32, p.38]. In the “real world” (i.e., outside the boundaries of TREC-like evaluation paradigms), each exploratory search is conducted by an individual person or a small group of people. In this sense, the process of conducting an exploratory search is inherently subjective. This inherent subjectivity can be addressed in many different ways. In some research traditions, the subjective experience is highlighted as the preferred focus of a particular social phenomenon. In these traditions, the researcher attempts to understand the phenomenon of interest from the participants’ perspectives, seeing it through their eyes. This approach rests on the assumption that reality is, in fact, socially constructed and cannot be understood apart from the understandings of those participating in its construction [5]. However, for others (includeing most system designers), subjectivity introduces “fuzziness” or complexity into our understanding of a particular reality [27, p.5], making it difficult to generalize study findings beyond the study participants. The subjectivity inherent within a particular search trajectory has implications both for understanding the search process and for evaluating the effectiveness of a particular IR system in supporting the search process. Most importantly, the inherent subjectivity of the search process may work against both the standardized evaluation of a system’s effectiveness and, possibly, even the design of systems that are useful to a large number of people. This paper will explore the implications of subjectivity at three points in an exploratory search: the starting point for the search, the trajectory of the search being undertaken, and the conclusion of the search. In addition, implications for evaluation methods and for system design will be discussed. THE STARTING POINT FOR THE SEARCH A variety of individual characteristics may affect a search at its beginning, but perhaps the most prominent of these is the domain knowledge of the searcher. The focus of exploratory search on learning and investigation makes it very likely that the searcher’s domain knowledge will influence retrieval success, as well as the search process. Somewhat surprisingly, domain knowledge has not been consistently found to affect retrieval success. Studies finding a positive effect of domain knowledge on retrieval success include those by Jacobson and Fusani [19] and Marchionini [22]. However, a number of studies have found no such relationship [1, 4, 13, 26, 33]. Thus, whether domain knowledge affects the recall and precision of a retrieved set is still an open question. A few studies have looked directly at the effect of domain knowledge on the process of searching, identifying the effects of domain knowledge on the tactics used in search strategy formulation and reformulation [34], the amount of time spent monitoring a search and the frequency with which terms were combined [18], the focus/attention of the searcher and his or her expectations for the outcomes of a search [23], and the searcher’s ability to choose appropriate